This tutorial will walk through how to setup a Virtual Breadboard test circuit to send 2 axis gesture analog data to a Raspberry Pi Pico via the EDGEY interface for classification. The classified label is output on an IO pin which is read back into Virtual Breadboard and displayed as a LED and also linked to a graphic card displaying the result visually.

Prerequisites

The test circuit consists of a ScratchPad which generates X,Y voltages from 2D drawn gestures and a DIP component which is used to set the labels being gestures being captured.

Components Used

Place Scratch pad

The Scratch Pad Pad component captures mouse or stylus motions into X,Y voltages that can be recorded as gesture motion.

  1. Click the Toolbox ribbon
  2. From the Tiny ML group click the ScratchPad icon to begin placing
  3. Position and click to finalise the position
  4. Click again to select and properties and configure as

Property

Value

Mode

record-playback

Period

2

Place Raspberry Pi Pico

The Raspberry Pi Pico avatar synchronises IO with a physical Pico device via the EDGEY interface. The test circuit will be sending analog gesture data to be read into the Pico and classified with the neural network model programmed in the Pico and will read the IO output and display the results in VBB.

  1. Click the Toolbox ribbon
  2. From the Avatar group click the Pico icon to begin placing
  3. Position and click to finalise the position
  4. Click again to select and properties and configure as

Property

Value

Connection

Select COM Port EDGEY is connected to

Place Sample and MUX Component

The Sample and Mux component is used to send multiple analog channels over a single analog DAC channel.

  1. Click the Toolbox ribbon
  2. From the Tiny ML group click the Sample and MUX icon to begin placing
  3. Position and click to finalise the position

Wire the components together using the following netlist

The Basic Output circuit consists of a LEDN component used to show visually the current state of an IO pin. There are 3 pins, Rock, Paper, Scissors used to show the current state. Place a LEDN, change it's property to 3 pins and positions to GPIO2,3,4 of the Pico.

  1. Click the Toolbox ribbon
  2. From the LED's and Optronics group click the LEDN icon to begin placing
  3. Position and click to finalise the position
  4. Click again to select and properties to a 3 pin
  5. Press space or rotate icon to rotate to a vertical orientation
  6. Move to position over GPIO2,3,4

Property

Value

PinCount

3

You are now ready to test the gesture classifier.

  1. Click Power ON to run the test circuit
  2. Draw the Rock Gesture - you should see the GPIO2 turn on
  3. Draw the Paper Gesture - you should see the GPIO3 turn on
  4. Draw the Scissors Gesture - you should see the GPIO4 turn on

Rock

Paper

Scissors

While classifying the Pico is outputing additional debug information regarding the inference results of the neural network used to reach it's conclusion

  1. Open a Serial Terminal such as Tera Terminal or HyperTermina
  2. Open a Serial Connection with the Pico
  3. View the Inference probabilities as you perform the different gestures

While LED's give a quick visualisation of the result it is often more satisfying to build a rich visual control panel. The Toggle Image component can be used to quickly build a visual control panel using screenshots captured using the Windows Snipping Tool.

  1. Click the Toolbox ribbon
  2. From the LED's and Optronics group click the Toggle Image icon to begin placing
  3. Position and click to finalise the position
  4. Click again to select and property Pin = top to position the pin to the top

Capture Screenshots to setup the Toggle Images

In the next steps the Snipping Tool is used to clip screenshot regions and save them as images. We are going to take snapshots of the Scratch Pad drawings to use them as control panel elements. We need 4 images, a blank image with no drawing which all 3 Toggle Image components will use and drawn image of each of Rock, Paper, Scissor gestures.

Capture the Blank ScratchPad Image

  1. Press Power On to start the application which will clear ScratchPad of any drawing
  2. Use Snipping Tool to capture the blank Scratch Pad image
  3. Press Power Off to return to edit mode

Paste to Toggle Image Bitmap OFF

The Image Property Editor custom dialog allows you to paste from the Clipboard to an image property

  1. Select the Toggle Image component
  2. Select the BitmapOff custom dialog [...]
  3. Select ‘Paste from Clipboard' option

Duplicate the Toggle Image

Duplicate 2 copies of the Toggle Image with the empty Bitmap Off and move them side by side

Capture Image of Rock Gesture

  1. Press Power On to start the application which will clear ScratchPad of any drawing
  2. Use Snipping Tool to draw a Rock Gesture
  3. Wait for the Gesture to be redrawn as Red
  4. Use Snipping tool to capture the ScratchPad with the drawn gesture
  5. Press Power Off to return to edit mode
  6. Select the BitmapMap custom dialog [...] of the leftmost Toggle Image component
  7. Select ‘Paste from Clipboard' option

Capture Image of Paper and Scissors Gesture

Repeat the previous step for both Paper and Scissors. You wan't see the captured gesture because they are saved to the BitmapOn property

Wire Up the Visual Panel to the GPIO

The Toggle Image's can now be wired to the GPIO Pins.

Now when you draw the gestures the classification will display the image of the performed gesture giving satisfying feedback and also the basis for a gesture based game.

  1. Press Power On to start the application which will clear ScratchPad of any drawing
  2. Draw each of the Rock, Paper, Scissors gestures
  3. Confirm the gesture is correctly classified and visualisation panel is displayed

Congratulations, you have successfully created a test circuit, accurately classified 3 gestures and created a visual display panel that displays a picture of the gesture you performed all without any programming.